Skip to content

feat(core): expose aiohttp pool wait telemetry - #3632

Merged
ananthsub merged 30 commits into
mainfrom
athitten/issue-3008
Oct 1, 2026
Merged

ananthsub merged 30 commits into
mainfrom
athitten/issue-3008

Conversation

@athitten

@athitten athitten commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Closes #3008.

  • Adds queued-only queue_duration_ms and all-attempt connect_total instruments under gym.http.connection_pool, labelled with nemo.gym.http.destination.server.name.
  • Uses QueueTimedTCPConnector only during metrics export; otherwise plain TCPConnector. No pool TraceConfig overhead.
  • Queued redirects/retries get separate samples; immediate acquisitions get none. Outcomes: ok/abandoned.
  • Reports per-server configured/effective limits once, validates all workers, and warns when pools or host capacity are below declared demand.
  • Preserves 0 = unlimited, rejects positive limits that divide to zero, and routes remote-agent POSTs through the shared wrapper without replaying them.

Validation

Acceptance criteria:

  • Queue duration is separate from HTTP CLIENT-span duration.
  • Histogram count/connect_total distinguishes queued/immediate acquisitions; constraints: total, per_host, unknown.
  • Startup reports configured per-server budgets and effective worker limits.
  • Undersized declared-demand budgets warn; invalid worker division fails.
  • Disabled metrics use the plain connector with negligible measured immediate-request overhead below.
  • Tests cover one/multiple workers, destinations, unlimited limits, and integer division.
  • Retries, tracing, cookies, redirects, cancellation, and response behavior are preserved.

Real-connector regressions cover timeout/retry, cancellation, queued redirects, first-request/ServerClient labels, CLIENT spans, and aiohttp method/signature compatibility. Independent reviews checked code/tests and benchmark methodology/results.

Checks on e74b8e865:

  • .venv/bin/pytest tests/unit_tests/test_server_utils.py tests/unit_tests/telemetry tests/unit_tests/test_opensandbox_http_transport.py tests/unit_tests/test_fern_docs_links.py -q -o addopts='' -o log_cli=false — 398 passed, 1 skipped, 17 subtests passed.
  • .venv/bin/pre-commit run --all-files — passed.
  • git diff --check — passed.
  • cd fern && npm run check -- --warnings — 0 errors; authenticated redirect checking was skipped without a Fern token.

Rollouts

N/A for model compute: shared HTTP observability/transport changes, not model or verifier decisions. Real aiohttp services exercise the affected request paths.

Compatibility and benchmark impact

No config/dataset migration; 0 stays unlimited. Budgets apply per server/CLI process group, not deployment-wide. Export must precede client creation; existing clients are not retrofitted. Dashboard/log queries should use the destination attribute and per_worker_per_host field. Histogram percentiles are queued-only.

The issue benchmark plan was rerun on e74b8e865: 10 ms loopback backends, concurrency 128/256/512, 1/4/16 client-worker processes, all three pool-sizing cases. Four interleaved settings (off, CLIENT spans only, metrics only, both), five repetitions: 540 cells, 1,873,920 measured requests, zero errors. Recorded queue/client-span/backend p50/p95/p99, throughput, sockets, FDs, errors. Warmup excluded; counter deltas matched requests. Total/adequate cases used four host/port destinations; per-host used one.

Four workers, concurrency 128, metrics+CLIENT spans enabled. Worker total/per-host limits: adequate 64/64, per-host 32/8, total 8/32. Latencies: ms, medians of five round-specific percentiles.

Pool sizing Queued-only p50/p95/p99 CLIENT span p50/p95/p99 Backend p50/p95/p99 Requests/s Peak sockets/FDs Queued fraction
Adequate — (no samples) 11.09/11.97/12.35 10.30/10.97/11.12 10,730 136/196 0.00%
Per-host undersized 37.81/49.44/49.85 50.24/61.85/62.38 10.91/11.34/11.46 2,635 40/100 90.08%
Total undersized 120.51/228.72/233.79 11.25/124.95/237.55 10.64/11.31/11.94 3,689 78/138 5.08%

Sockets/FDs sum worker peaks sampled every 5 ms, not simultaneous deployment peaks. Raw queue capture supplements SDK recording here; CPU is measured separately.

Throughput ranges: 6,927–10,908 / 2,546–2,802 / 3,273–3,717 requests/s respectively. One adequate round had CLIENT-span p99 71.87 ms. Total-limit queueing ranged 4.77–5.86%, making all-request p95 vary 12.65–125.91 ms. aiohttp's total limit is a soft scheduling limit, not a strict acquired-connection/socket cap across destinations; this case measures intermittent waits, not steady queueing behind a hard cap.

Isolated latest-branch CPU comparison: six interleaved rounds, concurrency 32, separate pinned client/backend cores, real SDK/in-memory reader; no CLIENT spans, queue observer, or FD sampler. Immediate requests used 8,192 measured + 1,024 warmup; undersized cases 2,048 + 256 with 10 ms backends. 54 cells, 221,184 measured requests, zero errors. CPU ranges are round minima/maxima; p95 covers full response reads.

Workload Metrics off CPU µs/request (range) Metrics on CPU µs/request (range) Off/on requests/s Off/on request p95 ms
Immediate, adequate pool 58.39 (57.77–60.46) 60.12 (59.73–60.85) 17,117/16,633 1.72/1.77
Per-host undersized 150.54 (149.77–151.93) 173.54 (171.36–174.16) 700/695 61.90/67.75
Total undersized 158.50 (153.61–168.57) 160.44 (145.32–171.50) 976/976 11.22/11.26

Matching-main baseline 12399ad81: immediate CPU 58.21 (57.67–58.41) µs/request. PR metrics-off median is +0.30% with overlapping ranges, not proof of zero cost. Enabled CPU cost: +2.96% immediate; +15.28% per-host (71.17% queued); +1.22% total, where only 0.78% queued (0.63–0.98%). The total case is not sustained-queue cost. No universal throughput/latency improvement is claimed.

Environment: AMD Ryzen Threadripper 3960X, Linux, Python 3.13.14, aiohttp 3.14.1, OpenTelemetry SDK 1.44.0. Network exporter/periodic collection, DNS/TLS/proxies, model throughput, and memory/allocation costs were not measured. Benchmark scripts and raw evidence remain local and are not committed.

Checklist

  • I have read the contributing guidelines.
  • The change is focused.
  • Tests added or updated pass the focused local checks above.
  • Documentation added or updated.
  • Pre-commit checks pass locally.
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off.

Signed-off-by: Abhishree <athittenaman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@athitten athitten added area:core Shared APIs, servers, telemetry, health, and registries feature New capabilities, enhancements, or enablement work labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

@athitten athitten self-assigned this Sep 22, 2026
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Signed-off-by: Abhishree <athittenaman@nvidia.com>
Signed-off-by: Abhishree <athittenaman@nvidia.com>
@athitten

Copy link
Copy Markdown
Contributor Author

/ok to test 9b3f3d2

Signed-off-by: Abhishree <athittenaman@nvidia.com>
@athitten

Copy link
Copy Markdown
Contributor Author

/ok to test 6c46eae

Signed-off-by: Abhishree <athittenaman@nvidia.com>
@athitten

Copy link
Copy Markdown
Contributor Author

/ok to test db5d90e

@athitten
athitten marked this pull request as ready for review September 23, 2026 22:12
@yaoyu-33 yaoyu-33 added complexity:medium Single-domain change with interacting parts or a moderate review surface needs-review PR is ready for code review and waiting on a reviewer labels Sep 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 1, 2026
Signed-off-by: Abhishree <athittenaman@nvidia.com>
@athitten

athitten commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d6b15ff

ananthsub
ananthsub previously approved these changes Oct 1, 2026
Signed-off-by: Abhishree <athittenaman@nvidia.com>
@athitten

athitten commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test a8c03d7

@ananthsub
ananthsub added this pull request to the merge queue Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/36940078210

Merged via the queue into main with commit 3117992 Oct 1, 2026
44 checks passed
@ananthsub
ananthsub deleted the athitten/issue-3008 branch October 1, 2026 23:32
khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 13, 2026
## What does this PR do?

Follow-up to NVIDIA-NeMo#3632, split out as requested in
NVIDIA-NeMo#3632 (comment).

In `_request_with_retries()`, the generic `except Exception` branch
handles every error except `ServerDisconnectedError` and
`ClientOSError`, which have their own branches. This includes aiohttp
timeouts. When the caller does not pass `_max_num_tries`, this branch
counted tries only for external callers. So an internal caller
(`_internal=True`) that set `_max_connection_retries` to 2 or more never
reached the cap and retried forever.

NVIDIA-NeMo#3415 already counts tries for callers that pass `_max_num_tries`. This
PR fixes the remaining path, where `_max_num_tries` is not passed.

- Count tries for internal callers too, so `_max_connection_retries`
stops them.
- Keep everything else the same:
- internal callers with no cap still retry with no limit and print no
retry message, as NVIDIA-NeMo#90 intended;
  - external callers still stop at `MAX_NUM_TRIES`;
  - callers that pass `_max_num_tries` are unchanged.
- Update the comment that said internal calls are not counted. It keeps
NVIDIA-NeMo#90's reason for not limiting internal calls.
- Add three tests:
  - the cap stops internal callers, with tracing off and on;
  - internal callers with no cap retry silently past `MAX_NUM_TRIES`;
- external callers stop at `MAX_NUM_TRIES` and print one retry message
per try.
- The cap test, `test_request_retry_cap_bounds_internal_callers`,
replaces `test_retry_cap_bounds_internal_callers`, which was removed
from NVIDIA-NeMo#3632.
- Make the retry tests fail instead of hang when the loop never stops:
- The test helper `_mock_global_client` (added in NVIDIA-NeMo#2383) now takes the
error to raise.
- Its fake `request()` calls `pytest.fail()` once it is called more
often than any test needs. `pytest.fail()` raises a `BaseException`,
which the loop's `except Exception` does not catch. So the test fails at
once, even if the loop never sleeps.
- Two NVIDIA-NeMo#3415 tests now use this helper instead of their own mocks:
`test_explicit_transport_attempt_cap_disables_request_retry` and
`test_request_caps_generic_errors_at_the_lower_attempt_limit`. Before,
if the generic branch ignored `_max_num_tries`, the first test hung,
sleeping a real 0.5 s per retry.

No caller in this repo sets both `_internal=True` and
`_max_connection_retries` today:
- The internal callers pass no cap: `ServerClient`, the simple agent's
direct tool calls to Resources servers (`tool_access`), the Terminus 2
sandboxed agent's model client (`NeMoGymAsyncOpenAI(internal=True)`),
the OpenHands capture route, and the LiteLLM version check.
- The callers with a cap are not internal: the remote agent and sandbox
transport use 1, Tavily search uses 0, and vLLM with `endpoint_file`
uses a default of 8.
- The OpenAI model server and the Gymnasium agent pass `_max_num_tries`
only when `upstream_max_num_tries` or
`model_server_transport_max_attempts` is set. Neither passes
`_max_connection_retries`, so neither changes.

Related:
- NVIDIA-NeMo#3553 added the cap check to this branch without counting internal
tries.
- NVIDIA-NeMo#3415 added `_max_num_tries` and counts tries on that path.
- NVIDIA-NeMo#3159 tracks bounding every retry by default, including internal
callers with no cap. This PR only makes an explicit cap work for them.
- NVIDIA-NeMo#2216 tracks the `ServerDisconnectedError` and `ClientOSError`
branches, which this PR does not change.
- Open PR NVIDIA-NeMo#2935 adds a timeout branch that copies the old counting, so
internal callers there still do not count tries. Whichever PR merges
later must count tries for every caller.

## Validation

All runs used an activated venv from `uv sync --extra dev --extra
sandbox --extra openshell --extra telemetry --extra mlflow --extra
wandb` (the extras in `full-test-suite.yml`). Every installed package
matches `uv.lock` (192 packages, including aiohttp 3.14.4, nemo-lens
0.2.0+b0f977d, and pytest 9.0.3).

- `pytest tests/unit_tests/test_server_utils.py
responses_api_agents/remote_agent/tests/test_app.py
tests/unit_tests/telemetry/`: 465 passed.
- With the old code restored,
`test_request_retry_cap_bounds_internal_callers` fails in under a second
with `Failed: retry loop did not stop` instead of hanging. It fails the
same way when the retry sleep is also removed.
- Each change below makes at least one test fail. Each run of
`test_server_utils.py` took about 3 s, and none hung:
  - the cap is ignored;
  - internal callers are limited by `MAX_NUM_TRIES`;
  - internal callers print;
  - external tries are double-counted;
  - external callers stop printing the retry message;
  - the `MAX_NUM_TRIES` raise is removed;
  - the retry sleep after a generic error is skipped;
  - the cap check ignores `_max_num_tries` attempts;
  - the generic branch ignores `_max_num_tries`;
  - the traced path drops `_internal` or `_max_connection_retries`.
- `pytest -n 16 tests/unit_tests
responses_api_agents/remote_agent/tests`: 8089 passed, 5 skipped. The
per-test results match `main` at a3cf403 (8085 passed, 5 skipped). The
4 extra results are the new tests, with the cap test run twice (tracing
off and on).
- The same focused tests pass on a local merge with the latest `main`
(NVIDIA-NeMo#4300), which changes no package versions.
- `pre-commit run --all-files`: passed.

## Rollouts

N/A. This changes a retry counter in the shared HTTP helper. No caller
in this repo changes behavior, and no model, agent, environment, or
verifier logic is involved.

## Compatibility and benchmark impact

None for callers in this repo. A caller that sets `_internal=True` and
`_max_connection_retries` to 2 or more, without `_max_num_tries`, now
stops after that many tries on errors that reach the generic `except
Exception` branch, including timeouts, instead of retrying forever. This
includes code outside this repo that uses
`NeMoGymAsyncOpenAI(internal=True, max_connection_retries=N)`.
`ServerDisconnectedError` and `ClientOSError` keep their own count, as
before.

## Checklist

- [x] I have read the [contributing
guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md).
- [x] The change is focused; unrelated "drive-by" edits are tracked as
separate issues/PRs.
- [x] Tests added or updated and pass locally, or N/A for docs-only /
non-code changes (so CI unit/server checks pass when applicable).
- [x] Documentation added or updated, or N/A with justification. N/A:
internal helper behavior with no user-facing docs.
- [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so
CI lint/format checks pass).
- [x] New source files include the required Apache-2.0 SPDX header. N/A:
no new files.
- [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check
passes).

---------

Signed-off-by: Abhishree <athittenaman@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — a8c03d75 Deployed Oct 1, 2026 by copy-pr-bot[bot] via release / finalize / notify #4141
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose HTTP connection-pool wait time and validate pool capacity

3 participants